Flexible touch sensing system and method with deformable material

ABSTRACT

A single volume soft sensor capable of sensing real-time continuous contact and stretching. An electrical impedance tomography (EIT) technique with support vector machine (SVM) learning is employed to estimate changes of resistance distribution on the sensor caused by fingertip contact even during sensor deformation events. A deformation switch is incorporated to maintain the localization during deformation events.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/719,540 filed Aug. 17, 2018, which is hereby incorporated byreference in its entirety.

BACKGROUND

Stretchable soft sensors have been explored as promising input methodsfor adding interactions on both rigid and elastic physical objects,smart textiles, shape-changing surfaces, humanoids, and the human body.With a high flexibility and stretchability of the sensors, a wide scopeof natural applications have been suggested. Still, the expensive andmulti-step fabrication processes hinder production of inexpensive,customized soft sensors. Moreover, such sensors often cannot maintainlocalization of a sensing point when the material is deformed during atouch interaction. Therefore, improvements are needed in the field.

SUMMARY

By jointly emphasizing fabrication, multi-modality and novelcomputational methods, the present disclosure provides a single-volumesoft-matter sensor that provides multimodal sensing and is able tosupport and restore contact localization upon and after deformation ofthe sensing material. The presently disclosed sensor and associatedmethods allow users to fabricate sensors inexpensively, customizeinterfaces easily, and deploy them instantly for continuous touch input.

In certain embodiments, the presently disclosed device utilizescarbon-filled liquid silicone rubber, a non-toxic piezoresistiveelastomer material. The major hurdle in employing the carbon-filledsilicone as an interaction input is the lack of real time sensingcapability. This is mainly due to a rebound elasticity of the material,which causes a slow-recovery of the sensing signals after the materialdeformations that occur during an input event. In the presentdisclosure, an adaptive multi-sensing process is implemented using anelectrical impedence tomography (EIT) process to achieve real-timecontact localization and a learning-based support vector machine (SVM)to achieve deformation awareness. The disclosed system is therefore ableto update contact localization in the presence of deformation of thesensing material.

By employing the EIT and SVM technique, the presently disclosed systemenables a human touch to interface and interact with the sensor viaelectrodes placed on the material boundary only. In this way, the sensorcan be fabricated in a single-volume manner and implemented withoutinvasive wirings or electronics or other elements which have to befabricated and placed in the interior of the material boundary. Nointerior elements are required, instead the material itself is used as asensor. Using the disclosed method provides sensing contact localizationand stretching within the sensor material. To this end, users areallowed to perform interactions instantly after deployment without anyextra training processes.

According to various aspects, a system is provided, comprising a singlevolume soft sensor capable of sensing real-time continuous contact andstretching. A low-cost and an easy way to fabricate such piezoresistiveelastomer-based soft sensors for instant interactions is also provided.An electrical impedance tomography (EIT) technique with SVM learning isemployed to estimate changes of resistance distribution on the sensorcaused by fingertip contact and determine contact localization evenduring a material deformation event. The EIT image reconstruction isprocessed with a difference in resistance measurement (δV) which thedifference between an instant measurement reading (V_(i)) and ahomogeneous baseline reading (V_(H)). A deformation switch value isdetermined to maintain and restore the contact localization during andafter the deformation event. When deformation occurs, the most recent δVbefore the deformation event occurred is maintained and used during thedeformation event. Upon release from the deformation, we updated thehomogeneous baseline using δV, where V_(H)=V_(i)−δV. Using the presentlydisclosed method, the contact localization can be maintained during thedeformation and restored after the deformation as shown in FIG. 2B.

This summary is provided to introduce the selection of concepts in aform that is easy to understand the detailed embodiments of thedescriptions. The embodiments are then brought together in a finalembodiment which described an environment, thereby stressing that eachof the embodiments may be viewed in isolation, but also the synergiesamong them are very significant. This summary is not intended toidentify key subject matter or key features or essential featuresthereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features, and advantages of variousexamples will become more apparent when taken in conjunction with thefollowing description and drawings wherein identical reference numeralshave been used, where possible, to designate identical features that arecommon to the figures, and wherein:

FIG. 1A illustrates a change in sensor output values without dynamicmanipulation upon a deformation event.

FIG. 1B illustrates a change in sensor output values with dynamicmanipulation upon a deformation event.

FIG. 2A illustrates a contact localization output for a stretching eventwithout a deformation switch value incorporated.

FIG. 2B illustrates a contact localization output for a stretching eventwith a deformation switch value incorporated.

FIG. 3A illustrates a sensor activating the initial sensing electrodepair.

FIG. 3B illustrates a sensor activating a subsequent sensing electrodepair.

FIG. 4 is a process flowchart illustrating a touch sensing processaccording to one embodiment.

FIG. 5 is a process flowchart illustrating a baseline update processaccording to one embodiment.

FIG. 6 shows a schematic view of a 16-electrode system.

FIG. 7 illustrates a high-level diagram showing the components of asensing system.

DETAILED DESCRIPTION

The term “drawings” used herein refers to drawings attached herewith andto sketches, drawings, illustrations, photographs, or other visualrepresentations found in this disclosure. The terms “I,” “we,” “our” andthe like throughout this disclosure do not refer to any specificindividual or group of individuals.

Sensing performed by the presently disclosed system is based on an EITtechnique with an SVM learning process which estimates the resistancedistribution of the conductive material using inverse problem analysisbased on measurements from the sensor boundary. The difficulty ofproviding real-time sensing with carbon-filled silicone rubber is due tothe material's rebound elasticity (>50%), which causes a long settlingtime (>10 s) and small shifts in baseline values as shown in FIGS. 1Aand 1B. FIG. 1A illustrates a change in sensor output values withoutdynamic manipulation upon a deformation event. FIG. 1B illustrates achange in sensor output values with dynamic manipulation upon adeformation event.

The presently disclosed sensing method is based on carbon-filled liquidsilicone rubber that changes its resistance distribution upon mechanicaldeformations. In one example, four-terminal sensing is used to measureresistance since this method reduces the inaccuracy from contactresistances. Unlike matrix tactile sensors where arrays of electrodesare required within the sensing area, the presently disclosed systemutilizes sensing electrodes 204 and a capacitive channel 206 coupled tothe outer edge of the sensor 202. Then, a Neighboring Method is usedwhere DC current is fed through two adjacent electrodes 202 and thevoltage differential is measured successively throughout the adjacentelectrode pairs as shown in FIGS. 3A and 3B. FIG. 3A shows the initialsensing electrode pair and FIG. 3B shows the next successive electrodepair sensing.

According to one embodiment, EIT image reconstruction is carried out bycomparing the measurements at two different instances. The update methodcomprises the following steps as shown in FIG. 4:

Raw sensor readings from sensors 204 are fed into EIT channel 404 andSVM channel 406.

Before feeding the sensor values to the SVM classifier (block 408),differential dynamic manipulation (block 410) is applied when the sensorsettlement enters the quasi-steady state, i.e.,dV_(avg)<dV_(avg,threshold) when V_(avg)<V_(avg,threshold).

The deformation type is classified and assigned a deformation identifier(block 413) using SVM with polynomial kernel (block 412).

If there is “No Deformation,” (block 414) the presence of deformation isconfirmed in the previous frame (block 416).

If the deformation event exists in the previous frame, V_(H)=V_(i)−δV isset to update the homogeneous baseline (block 418). Otherwise, EITlocalization (block 420) is processed using the baseline process 500 ofFIG. 5 to update δV (block 422) with the current Vi to localize acontact coordinate location in the sensor 202 (block 424).

If any deformation is detected, multiple channels are activated: 1) δVfrom the most recent localization during “No Deformation” is used (block426) and a contact coordinate is outputted and 2) the system determinesthe level of the corresponding deformation (block 430) using SVMregression (block 428) with a polynomial kernel, with the regressedvalues mapped to corresponding deformation (block 432).

FIG. 5 shows a flowchart 500 which illustrates a baseline updateprocess. First, if a contact is not detected, i.e., the capacitivesensing value cap, is less than a predetermined thresholdcap_(threshold) (stage 502), instant measurement readings (Vi) are setas a homogeneous baseline data (V_(H)) (stage 504). Ifcap_(i)>=cap_(threshold), a movement detection is evaluated (stage 506).If div Vavg,i>=div Vavg threshold, the system sets the previous frame'sdata (Vi−1) as VH (stage 508) and proceeds to perform an imagereconstruction using EIT (stage 510). If div Vavg,i<div Vavg threshold,the system directly proceeds to stage 410 and performs an imagereconstruction using EIT. The system may optionally apply a color filterto the reconstructed image for blob detection and localize a contactcoordinate from the center of the blob (stage 512) before outputting thecontact position (x, y).

Throughout this description, some aspects are described in terms thatwould ordinarily be implemented as software programs. Those skilled inthe art will readily recognize that the equivalent of such software canalso be constructed in hardware, firmware, or micro-code. Becausedata-manipulation algorithms and systems are well known, the presentdescription is directed in particular to algorithms and systems formingpart of, or cooperating more directly with, systems and methodsdescribed herein. Other aspects of such algorithms and systems, andhardware or software for producing and otherwise processing signals ordata involved therewith, not specifically shown or described herein, areselected from such systems, algorithms, components, and elements knownin the art. Given the systems and methods as described herein, softwarenot specifically shown, suggested, or described herein that is usefulfor implementation of any aspect is conventional and within the ordinaryskill in such arts.

FIG. 7 is a high-level diagram showing the components of the exemplarysystem 1000 for analyzing the EIT location data and performing otheranalyses described herein, and related components. The system 1000includes a processor 1086, a peripheral system 1020, a user interfacesystem 1030, and a data storage system 1040. The peripheral system 1020,the user interface system 1030 and the data storage system 1040 arecommunicatively connected to the processor 1086. Processor 1086 can becommunicatively connected to network 1050 (shown in phantom), e.g., theInternet or a leased line, as discussed below. The EIT data may bereceived using sensor 202 (via electrodes 204) and/or displayed usingdisplay units (included in user interface system 1030) which can eachinclude one or more of systems 1086, 1020, 1030, 1040, and can eachconnect to one or more network(s) 1050. Processor 1086, and otherprocessing devices described herein, can each include one or moremicroprocessors, microcontrollers, field-programmable gate arrays(FPGAs), application-specific integrated circuits (ASICs), programmablelogic devices (PLDs), programmable logic arrays (PLAs), programmablearray logic devices (PALs), or digital signal processors (DSPs).

Processor 1086 can implement processes of various aspects describedherein. Processor 1086 can be or include one or more device(s) forautomatically operating on data, e.g., a central processing unit (CPU),microcontroller (MCU), desktop computer, laptop computer, mainframecomputer, personal digital assistant, digital camera, cellular phone,smartphone, or any other device for processing data, managing data, orhandling data, whether implemented with electrical, magnetic, optical,biological components, or otherwise. Processor 1086 can includeHarvard-architecture components, modified-Harvard-architecturecomponents, or Von-Neumann-architecture components.

The phrase “communicatively connected” includes any type of connection,wired or wireless, for communicating data between devices or processors.These devices or processors can be located in physical proximity or not.For example, subsystems such as peripheral system 1020, user interfacesystem 1030, and data storage system 1040 are shown separately from thedata processing system 1086 but can be stored completely or partiallywithin the data processing system 1086.

The peripheral system 1020 can include one or more devices configured toprovide digital content records to the processor 1086. For example, theperipheral system 1020 can include digital still cameras, digital videocameras, cellular phones, or other data processors. The processor 1086,upon receipt of digital content records from a device in the peripheralsystem 1020, can store such digital content records in the data storagesystem 1040.

The user interface system 1030 can include a mouse, a keyboard, anothercomputer (connected, e.g., via a network or a null-modem cable), or anydevice or combination of devices from which data is input to theprocessor 1086. The user interface system 1030 also can include adisplay device, a processor-accessible memory, or any device orcombination of devices to which data is output by the processor 1086.The user interface system 1030 and the data storage system 1040 canshare a processor-accessible memory.

In various aspects, processor 1086 includes or is connected tocommunication interface 1015 that is coupled via network link 1016(shown in phantom) to network 1050. For example, communication interface1015 can include an integrated services digital network (ISDN) terminaladapter or a modem to communicate data via a telephone line; a networkinterface to communicate data via a local-area network (LAN), e.g., anEthernet LAN, or wide-area network (WAN); or a radio to communicate datavia a wireless link, e.g., WiFi or GSM. Communication interface 1015sends and receives electrical, electromagnetic or optical signals thatcarry digital or analog data streams representing various types ofinformation across network link 1016 to network 1050. Network link 1016can be connected to network 1050 via a switch, gateway, hub, router, orother networking device.

Processor 1086 can send messages and receive data, including programcode, through network 1050, network link 1016 and communicationinterface 1015. For example, a server can store requested code for anapplication program (e.g., a JAVA applet) on a tangible non-volatilecomputer-readable storage medium to which it is connected. The servercan retrieve the code from the medium and transmit it through network1050 to communication interface 1015. The received code can be executedby processor 1086 as it is received, or stored in data storage system1040 for later execution.

Data storage system 1040 can include or be communicatively connectedwith one or more processor-accessible memories configured to storeinformation. The memories can be, e.g., within a chassis or as parts ofa distributed system. The phrase “processor-accessible memory” isintended to include any data storage device to or from which processor1086 can transfer data (using appropriate components of peripheralsystem 1020), whether volatile or nonvolatile; removable or fixed;electronic, magnetic, optical, chemical, mechanical, or otherwise.Exemplary processor-accessible memories include but are not limited to:registers, floppy disks, hard disks, tapes, bar codes, Compact Discs,DVDs, read-only memories (ROM), erasable programmable read-only memories(EPROM, EEPROM, or Flash), and random-access memories (RAMs). One of theprocessor-accessible memories in the data storage system 1040 can be atangible non-transitory computer-readable storage medium, i.e., anon-transitory device or article of manufacture that participates instoring instructions that can be provided to processor 1086 forexecution.

In an example, data storage system 1040 includes code memory 1041, e.g.,a RAM, and disk 1043, e.g., a tangible computer-readable rotationalstorage device such as a hard drive. Computer program instructions areread into code memory 1041 from disk 1043. Processor 1086 then executesone or more sequences of the computer program instructions loaded intocode memory 1041, as a result performing process steps described herein.In this way, processor 1086 carries out a computer implemented process.For example, steps of methods described herein, blocks of the flowchartillustrations or block diagrams herein, and combinations of those, canbe implemented by computer program instructions. Code memory 1041 canalso store data, or can store only code.

Various aspects described herein may be embodied as systems or methods.Accordingly, various aspects herein may take the form of an entirelyhardware aspect, an entirely software aspect (including firmware,resident software, micro-code, etc.), or an aspect combining softwareand hardware aspects These aspects can all generally be referred toherein as a “service,” “circuit,” “circuitry,” “module,” or “system.”

Furthermore, various aspects herein may be embodied as computer programproducts including computer readable program code stored on a tangiblenon-transitory computer readable medium. Such a medium can bemanufactured as is conventional for such articles, e.g., by pressing aCD-ROM. The program code includes computer program instructions that canbe loaded into processor 1086 (and possibly also other processors), tocause functions, acts, or operational steps of various aspects herein tobe performed by the processor 1086 (or other processor). Computerprogram code for carrying out operations for various aspects describedherein may be written in any combination of one or more programminglanguage(s), and can be loaded from disk 1043 into code memory 1041 forexecution. The program code may execute, e.g., entirely on processor1086, partly on processor 1086 and partly on a remote computer connectedto network 1050, or entirely on the remote computer.

Those skilled in the art will recognize that numerous modifications canbe made to the specific implementations described above. Theimplementations should not be limited to the particular limitationsdescribed. Other implementations may be possible.

What is claimed is:
 1. A sensing system, comprising: a stretchable basematerial which changes its resistance distribution upon a mechanicaldeformation event; a plurality of electrodes attached to a perimeter ofthe base material; a capacitive sensing channel attached to the basematerial; and a control unit operatively connected to the plurality ofelectrodes and the capacitive sensing channel, the processing unitconfigured to utilize electrical impedance tomography with supportvector machine learning to estimate changes of resistance distributionon the sensor caused by a human body contact to determine areconstructed image of the location and shape of the human body contact,the processing unit further configured to evaluate when the mechanicaldeformation event is occurring and use a previously-stored reconstructedimage of the location and shape of the human body contact during saidmechanical deformation event.
 2. The sensing system of claim 1, whereinthe control unit is further configured to begin adaptively updating abaseline EIT measurement when the human body contact with the basematerial begins and stop updating the baseline EIT measurement when thehuman body contact with the base material ends.
 3. The sensing system ofclaim 2, wherein said human body contact beginning and ending are sensedby the control unit via the capacitive sensing channel
 4. The sensingsystem of claim 1, wherein the base material comprises a carbon filledelastomer.
 5. The sensing system of claim 1, wherein the control unitutilizes a neighboring method to sense the location and shape of saidhuman body contact, said neighboring method comprising: a) measuring afirst voltage differential between a first adjacent pair of theelectrodes; b) measuring a second voltage differential between a secondadjacent pair of electrodes, the first and second pairs of electrodeshaving a common electrode; and c) continuing to measure voltagedifferentials between successive neighboring pairs of the electrodesuntil all adjacent pairs of the electrodes have been evaluated for theirvoltage differential.
 6. The sensing system of claim 1, wherein thecontrol unit is further configured to: a) measure a baseline measurementof said voltage differential values; and b) determine when the basematerial has been mechanically stretched based on differences in thevoltage differential values.
 7. The sensing system of claim 1, whereinthe control unit is further configured to apply a color filter to thereconstructed image to localize a contact coordinate from the center ofthe image.
 8. The sensing system of claim 1, wherein the base materialis imprinted with graphics to indicate control buttons of a device usercontrol interface.
 9. The sensing system of claim 1, wherein theelectrodes are evenly spaced along the perimeter of the base material.10. The sensing system of claim 1, further comprising a current sourceconnected between the control unit and the electrodes.
 11. The sensingsystem of claim 1, further comprising an amplifier connected in a returnpath from the electrodes to the control unit.